Automated pharmaceutical pill identification

ABSTRACT

A pill identification system identifies a pill type for a pharmaceutical composition from images of the pharmaceutical composition. The system extracts features from images taken of the pill. The features extracted from the pill image include color, size, shape, and surface features of the pill. In particular, the features include rotation-independent surface features of the pill that enable the pill to be identified from a variety of orientations when the images are taken. The feature vectors are applied to a classifier that determines a pill identification for each image. The pill identification for each image is scored to determine identification for the pharmaceutical composition.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.14/373,596, filed Jul. 21, 2014, which is a national stage applicationof PCT/US2013/022754, filed Jan. 23, 2013, which claims the benefit ofU.S. Provisional Application No. 61/589,750, filed Jan. 23, 2012, theentire disclosure of each is hereby incorporated by reference herein inits entirety.

BACKGROUND

This disclosure relates generally to image processing, and morespecifically to identification of pharmaceutical compositions (e.g.,pills) by automated means.

Medication errors cause at least one death every day in the UnitedStates and injure approximately 1.3 million people annually. A Food andDrug Administration (FDA) study of fatal medication errors found thatthe most common errors involving medications were related to theadministration of an improper dose of medicine (41%) and administrationof the wrong medication (14%). Almost half of the fatal medicationerrors occurred in people over the age of 60, who often take multipleprescription medications. Such medication errors continue to occurdespite federal regulations implemented in 1995 that require imprintingof identification codes on all medication solid oral-dosage forms.

The task of administering medications to a patient in a hospital ornursing home environment remains a manual process with limited qualityassurance and a task that is highly subject to human error. Typically, anurse reads a patient's prescription, opens a bottle of pills with theintended medication, places the pills in a small unlabeled plastic cup,carries the cup to the patient's bedside, and directs the patient totake the pills in the cup. There is no independent quality assuranceprocess to confirm 1) that the correct medication and number of pillsare placed in the plastic cup, 2) that the medications are delivered tothe correct patient, or 3) that the medication is being administered atthe correct time (e.g., not more than every 4 hours).

Likewise, patients in a home environment shoulder a substantial amountof responsibility in managing their own medications, which can result inmedication errors. Common errors in the home include taking the wrongdosage or quantity of pills, forgetting to take certain medications ordoses, taking the medication at the wrong time, too many times a day, ornot enough times a day, among other problems. For patients takingmultiple medications a day or having medication regimes involvingcomplex timing and administration factors, careful day-to-day managementof medications can become quite difficult.

Errors in medications can also arise in the pharmacy environment. Filledprescriptions can be mislabeled with the incorrect dosage or amount ofpills, or with the incorrect medication. Pharmacists can dispense thewrong drug, quantity, or dosage, which are mistakes that can result inserious injury or death of the patient. Pharmacists can make these typesof mistakes as a result of being overworked or distracted, or even dueto confusion between medication names that are similar, or pills thathave similar physical appearances.

What is needed is a mechanism for quickly and easily identifyingmedication in an automated way to ensure the correct medication anddosage are provided. As part of an automated medication and dosagesystem, pills must be identified and confirmed as the intended pill fora patient. A particular challenge of automated identification of pillsis that a pill may be identified by any arbitrary markings chosen by apharmaceutical company. For example, pills may include arbitrary shapes,surface features, lettering fonts and orientation, color, and the like.These aspects of a pill's character provide particular challenges topill identification methods. In addition, solutions that require a pillto be oriented in a particular way for identification cause undue delayas a machine or operator orients the pill.

SUMMARY

A pill identification system is provided that identifies a pill byextracting features from images taken of the pill. The featuresextracted from the pill image include rotation-independent surfacefeatures of the pill that enable the pill to be identified in variousorientations as the image is taken.

A computer-implemented system, method, and computer-readable medium foridentifying a pharmaceutical composition are provided. At least oneimage of the pharmaceutical composition is accessed, wherein each imageprovides a distinct image of the pharmaceutical composition. Featuresare extracted from each image of the pharmaceutical composition, thefeatures including a rotationally-independent measure of surfacefeatures in the image. A feature vector is generated for each image ofthe pharmaceutical composition that includes the features extracted forthat image. At least one classifier is applied to each of the featurevectors to determine at least one classification associated with eachimage. A pill identification is determined by accumulating the at leastone classification for each image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 provides a conceptual view of an example pill verification systemaccording to an embodiment.

FIG. 2 illustrates an overview of a method for identifying a pill frompill images according to an embodiment.

FIG. 3 illustrates an overview of an embodiment of feature extractionand image classification according to an embodiment.

FIG. 4A illustrates a pill outline identification process according toan embodiment.

FIG. 4B illustrates another pill outline identification processaccording to an embodiment.

FIG. 5 illustrates a method for extracting features from a pill imageaccording to an embodiment.

FIG. 6 illustrates a method of extracting surface features from a pillimage according to an embodiment.

FIGS. 7-13 illustrate image processing techniques used for extractingsurface features according to an embodiment.

FIG. 14 illustrates a method for generating color components for a pillfeature vector according to an embodiment.

FIG. 15 illustrates a hierarchical classification methodology foridentifying feature vectors for pill images according to an embodiment.

FIG. 16 illustrates a method of pill identification using classifierscoring according to an embodiment.

FIG. 17 illustrates a pill identification system according to anembodiment.

The figures depict various embodiments of the present invention forpurposes of illustration only. One skilled in the art will readilyrecognize from the following discussion that alternative embodiments ofthe structures and methods illustrated herein may be employed withoutdeparting from the principles of the invention described herein.

DETAILED DESCRIPTION

(a) Overview

An image classification method and system for identifying pharmaceuticalcompositions (e.g., pills, which can include any type of pill, tablet,capsule, or other pharmaceutical composition) is provided according toan embodiment. The image processing method provides an identification ofpills by extracting features from images taken of a pill, applying ascoring algorithm to the extracted features, identifying a best-matchpill for each image from a pill identification database, and identifyingthe pill according to the most confident best-match pill from theidentified best-match pills.

In one embodiment, the image classification system is part of a pilldispensing and/or verification system. One example of such a system isdisclosed in International Patent Application No. PCT/US2011/027586,filed on Mar. 8, 2011, which claims priority to U.S. ProvisionalApplication No. 61/311,900, filed on Mar. 9, 2011, both of which arehereby incorporated herein in their entireties, however other systemsfor dispensing and/or verification can also be used or the imageclassification can be done independent of such a dispensing and/orverification system. In this example pill dispensing and verificationsystem, a pill is introduced into the system and the system capturesimages of the pill using cameras or other imaging devices placed aroundthe pill. The images are provided to the image classification system todetermine whether the pill images match the characteristics of pills ina database. Specifically, features are extracted from the pill images toform a feature vector to enable comparison of the images to features ofknown pills in a feature database. The identification of the pill isused by the system to determine, for example, whether the pill is partof a prescribed dosage for a patient.

FIG. 1 shows a conceptual view of an example pill verification system. Apill 100 travels down a tube 110 or pathway during the imaging process.The tube 110 is typically clear and enables cameras 120 to have anunobstructed view of the pill 100. The tube 100 may be configured to atleast partially orient the pill to reduce the variation among pillimages. In one embodiment, a square tube 100 is used at a slanted angleto encourage the pill to slide along a flat side of the pill, ratherthan freefall along any angle. Cameras 120 capture images of the pill100 as the pill passes through the tube 110, and the images areprocessed for pill identification. These images can be taken of the pillwhile the pill is moving through the system. The system does not requirethat the pill be stopped or held stationary while the images are taken,but instead can takes image of the pill in motion.

In one embodiment, the end of the tube 110 includes a gate 130 thatstops the pill until the identification is complete, such as while thepill images are being processed. Some embodiments do not include gate130. When the identification is complete, a diverter 150 is moved todirect the pill into either a positive or negative result bin 140. Thepill is diverted to the positive result bin 140 if the pill isidentified as a pill and dosage prescribed to the patient, and to thenegative bin 140 otherwise. Once the diverter is in place, the gate 130opens and the pill is directed to the appropriate result bin 140. Inother embodiments, other criteria determine whether to place the pill inthe positive bin. For example, in some configurations, any quantity of aspecific type of detected pill is diverted to the positive bin. Infurther embodiments, several result bins 140 are used, and each pilltype is diverted according to its identification. Several result binsmay be used, for example, to sort a mixture of different types of pills.Thus, a user adds pills to the verification system and receives a quickand automated determination of a pill's identity. Since the pill isidentified as the pill proceeds through the tube 110, the system is ableto identify and sort pills quickly. In other embodiments, positive andnegative bins are not used, but instead the system indicates whethereach pill has been positively or negatively identified (i.e., a positiveor negative match to a designated pill type) or otherwise provides anindication of or information about the pill type or identity.

(b) Pill Classification Overview

FIG. 2 illustrates an overview of a method for identifying 201 a pillfrom pill images according to one embodiment. This figure provides justone example of steps that may be included in the method, but the stepsmay vary or may be differently ordered in other embodiments. The same istrue for all methods described throughout. In FIG. 2, first the systemaccesses 202 images of a pharmaceutical composition (e.g., the pill),for example from cameras 120. Alternatively, pill images are accessed byreceiving pill images from a system separately from cameras 120. Thatis, the pill identification method described herein may be useful forany application that identifies a pill from one or more pill images.

In order to classify the pills, features are extracted 203 from each ofthe pill images. As described in detail below, the features extractedfrom the pills in this embodiment include color, size, shape, andsurface features, though any suitable features or set of features may beused. The extracted features are used to create a feature vector thatrepresents the pill image using the features associated with the pillimage. Next, each image from the camera is classified 204 by aclassification algorithm to identify the most likely pill represented inthe image. After classification of each image, the results from theclassifiers are scored 205 to merge results from the various images.Since the images may provide differing pill identifications, the scoring205 determines scores for each candidate pill type to select 206 thepill type with the highest score.

An overview of an embodiment of the feature extraction and imageclassification methodology (e.g., steps 203 & 204 of FIG. 2) isdescribed with reference to FIG. 3. The image classification starts 301with a raw image photograph or other image of the pill. In order toextract features from the image corresponding to the pill, the portionof the image containing the pill is identified. To identify the pill,the pill identification system determines the pill's outline and removesreflections in the image 302. Once the pill has been identified, featurevectors are created 303 from the pill image through a variety of imageprocessing techniques. That is, in this embodiment, each pill image isassociated with a feature vector representing the pill image. Thefeatures extracted from pill image are classified 204 by theclassification algorithm. The classification algorithm uses a variety ofclassification methods and can output a Boolean best-match pill type forthe image, or alternatively the classification method indicates abest-match pill type using a confidence value. The classifier may be anysuitable machine-learning classification system, such as a supportvector machine (SVM).

Prior to classifying feature vectors 304, the pill classifier is trainedon a plurality of images of known pills. Using a training process, avariety of training images is used that includes information about thecorresponding pill types for the training images. The pill classifier istrained on the features extracted from the training images to determinethe features that characterize each pill type. Though described as asingle pill classifier, the pill classifier can comprise a plurality ofclassifiers, where each classifier is trained on the set of features andpill types for that particular classifier. As described below, theclassifiers may be hierarchically organized, wherein a high-levelclassifier is used to discriminate among easily differentiated featurevectors. For example the high-level classifier may discriminate based oncolor, distinguishing pills as white, brown, red, blue, etc. Intraining, the pills may be designated by the operator as belonging aparticular type (e.g., color), or the feature vectors may beautomatically clustered. The feature vectors associated with the pilltypes in the clusters are used to train the high-level classifier. Afterclassification to a high-level group, a lower-level classifier mayfurther distinguish among additional clusters, or may classify thefeature vector as a pill type. In one embodiment, the classifiers aremaintained in a pill classification database.

(c) Pill Outline Detection

In one embodiment, prior to processing an image to extract features, theimage of the pill is processed to identify the location of the pillwithin the image and to orient the pill. This pill outlineidentification is used to clean up the image and ensure that thefeatures extracted from the image are features which originate from thepill and not from background noise or other aspects of the image. Thispill outline identification is also used because images may capture allor part of the pill located at various areas of the image and particularpill types have a variety of shapes and sizes.

One embodiment of a pill outline identification process is describedwith reference to FIG. 4A. The pill outline identification processstarts 401 by converting 402 the image to grayscale. Next, the grayscalerange of the image is dynamically scaled 403 to calibrate the colorrange of the original grayscale image to a full grayscale range. Inother words, if the original grayscale range uses a value range of 20-60in a processing range capable of 0-255, the value range 20-60 is scaledto use the entire 0-255 range. Next, edges contained within the pillimage are detected 404. This can be done using a Canny filter or anothermechanism for identifying the edges. The detected edges may compriseportion of the pill image with a sharp contrast from one portion toanother, such as the actual edge of the pill, but also surface marks,such as any lettering on the pill, other imprints on the pill, and otheraspects of the image with high contrast. Thus, the “edges” as describedin this portion do not initially make up the outline of the pill.

Continuing with this embodiment for pill outline identification, thedetected edges are now converted 405 to a contour list. Each contourcorresponds to an individual contiguous edge detected in the image. Thesegments are inspected and contours that fail to meet minimum thresholdsare removed 406 from the contour list. For example, contours may berequired to meet a minimum length and a minimum aspect ratio, such as anaspect ratio greater than 0.1, and a minimum area (i.e. the number ofpixels in the contour). Shadows and other noise can detected and removedin this way, which increases the probability that only the contours thatare a part of the pill outline are selected. Next, open-ended contoursare analyzed to determine if the opening of one contour matches theopening of another contour by identifying adjacent open-ended contoursand connecting 407 the adjacent contours. After connecting contours, thecontours now include enclosed spaces. The enclosed spaces which fail toenclose a minimum area are rejected 408. Next, the contours areconverted 409 into regions of interest. Another filtering step removes410 the regions of interest whose aspect ratio exceeds a minimumthreshold and regions that are too close to the edge of the field ofview. Phrased another way, regions of interest which are too close tothe edge of the pill or which are excessively elongated are assumed notto constitute portions of the pill for outline detection purposes.

The remaining regions of interest are now merged 411 into a singleregion of interest and any holes in the region of interest are filled412. Next, the region of interest is smoothed by applying 413 an openingfilter to remove any protrusions outside the outline of the region ofinterest. Finally, the region of interest is expanded to fill theinternal spaces. That is, any internal spaces are expanded 414 to theconvex hull of the region of interest. This creates the pill outlinethat may be used for extracting features of the pill.

Another method of identifying a pill outline is shown in FIG. 4Baccording to one embodiment. In this method, the pill outline isdetermined by two pill outline methods operating in parallel. The twomethods are compared to determine which method provided the betterresult and may be combined to generate the final pill outline. Afterstarting 430 the pill outline identification, a step identifies 431 tubeboundaries in the image. The image of the pill 100 taken by the cameras120 may include portions of the tube in the pill image. The tubeboundaries are typically identified in the upper and lower regions ofthe image, though the locations will vary based on the arrangement ofthe tube 110 and cameras 120. In one embodiment, the tube boundaries areidentified by lower-intensity regions in the corners of the image (upperleft/right and lower left/right). The location and angle of the tubeboundaries is identified and computed.

In the first outline determination method, a set of edges is identified440 in the image by identifying high-contrast areas in the image. Theedges may be detected by any suitable method, such as through a cannyfilter. Next, the edges identified in the image that are adjacent toeach other are joined 441. In one embodiment, the joined adjacent edgesare cotangent to one another. After joining, pill regions are createdfrom edges that form closed contours.

Next, reflective regions are identified and removed 460. Reflectiveregions are determined by comparing pill regions with the tubeboundaries and the edge of the image. Pill regions that are portions ofthe tube boundaries and regions that are predominantly near the edge ofthe image (e.g. the top and bottom) are removed from the pill regionlist. For the remaining pill regions, a count of background pixels isdetermined 461 that indicates the extent to which the pill regionsenclose pixels that are likely part of the pill background. Thebackground pixels are determined by identifying pixels that are dark inall color channels (i.e. in all color types). The threshold for darknessmay be determined based on the configuration of the tube 110 and cameras120. When the background pixel count is high, it indicates that the pillregions may be significantly larger than the pill, and indicates theremay have been an error in the pill outline determination. Finally, thepill regions are filled 462 to the convex hull of the pill region list.This combines the pill regions to produce a pill outline that removesnegative curvature inflections.

The second outline detection method starts with grayscale segmentation450 of two different mathematically derived images from the original RGBimage. The first grayscale image is created from the variation in thesaturation color channel. While the color may change in the pill fromregion to region, the saturation is typically relatively consistentthroughout the pill and noisy in background portions. The secondgrayscale image is an image of the maximum grayscale pixel value of eachof the RGB color channels. The segmentation results from each image arecombined into a single pill region. Next, pill reflections aredetermined by identifying reflection lines 451. The reflection lines usea convolution kernel to identify horizontal line pixels. The horizontalline pixels are identified in the tube boundaries to identify likelyportions of the reflection. The pixels from the convolution kernel arecombined to identify the reflection line 451. Using the reflection line,the region of interest generated by the grayscale segmentation 450 isbroken into multiple disconnected regions by separating the reflectionfrom the pill regions 452. The separation of the pill regions is basedon the reflection line. The pill regions associated with reflections areremoved 460, background pixels counted 461, and the regions are filledto the convex hull 462 as described above. Thus, two sets of regions ofinterest are created, the first through edge detection, and the secondthrough grayscale segmentation.

Next, the two pill outlines are combined 470. The outlines from theedge-based and grayscale-based outlines are used to determine a singlepill outline. The combined outline may be the pill outline from one ofthe two pill outlines, or may be a combination of the pill outlines. Thedetermination of which combined outline to use is determined based onthe background pixel count of each region of interest. When thebackground pixel count is significantly lower for one pill outlinerelative to the other, the lower pill outline is selected as thecombination pill outline. When both pill outlines have similar pixelcounts, the union of the regions of interest for each pill type is usedto determine the regions of interest in common to both. In addition,remaining inflection points may be identified by analyzing inflectionpoints in the regions of interest. The inflection points are examined todetermine if the inflection points make up a line. If such a line isidentified in the region of the image associated with the tube, theregion of interest associated with the line is removed from the pilloutline and the remaining regions of interest are returned as the pilloutline.

As a result of the foregoing methods, the outline for the pill isidentified and can be used for extracting features of the pill for usewith a pill classification method. The foregoing is one embodiment of apill outline identification method; other methods of identifying thepill's outline may be used in other embodiments.

(d) Feature Vectors

Using the identified pill shape from the image, features are extractedfrom the images and feature vectors are constructed to characterize thepill image. The feature vectors are used by a classification system,both to train classifiers by associating the pill images with a knownpill type, and by using the trained classifiers to identify an unknownpill. In one embodiment, the feature vectors comprise componentsextracted from the pill's shape and size, surface features of the pill,and color components of the pill.

An embodiment of a method for extracting features from a pill image isshown in FIG. 5. Feature vector creation begins 501 after determiningthe outline of the pill as discussed above. First, a rectangle isgenerated enclosing the pill outline and it is reduced in size until thesmallest rectangle to enclose the pill outline is determined 502, suchthat the rectangle may have any orientation around the pill outline.Next, the enclosed pill outline is rotated 503 so that the rectanglewith the pill outline is horizontal. That is, the rectangle is used torotate the pill to a predominantly horizontal position, such that aspectratio and other features are easily and consistently measured acrosspill images. Other methods of rotating the pill image to a horizontalposition may be used, or features are calculated in some embodimentswithout rotating the pill image. In this embodiment, features areextracted from the rotated pill image. As an overview, shape and sizecomponents are calculated 504, surface-feature components are calculated505, and color components are calculated 506 from the pill outline. Thegeneration of these various components is described in further detailbelow. These components are added 507 to a feature vector associatedwith the image.

In addition, features from the other images taken of the pill may beadded 508 to the feature vector and sorted in order to bettercharacterize some types of features. In this embodiment, size and colorfeatures from each image are be added to the respective portions of thefeature vector for each image. For example, a size-related area featureof each image is added to the area feature vector of each image. Priorto adding the area feature data, the data for the images is concatenatedand sorted. Thus, if three images include associated area features of19, 32, and 25, the area features are concatenated and sorted togenerate a set of area features {19, 25, 32} that is added to eachfeature vector for the three images. The addition of such data enablesthe classifier to train on data indicating, for example, that the imagewhich shows a short side of a pill is in fact the short side of agenerally oblong pill. The features that are added to each featurevector in this way are described more fully below, and includesize-based features of area, length, and aspect ratio. A featuredescribing the surface-feature complexity of each image may also beadded to each feature vector. This feature may indicate generally, forexample, the distribution of surface textures on various sides of thepill.

(e) Feature Vector—Shape and Size Components

Using the pill image and outline, one embodiment generates a variety ofshape and size components for the feature vector. The size and shapecalculations include, for example, the length of the longest dimension,the length of the pill outline, the area, the aspect ratio, thecircularity, the compactness, convexity, bulkiness, and structurefactor.

The circularity measures the similarity of the pill outline to a circle,and is calculated in one embodiment by calculating the area of theregion and dividing the area by the maximum distance from the center tothe contour pixels squared, multiplied by pi(Circularity=(area)/(maximum distance)₂*π).

The compactness measures the length of the contour relative to the areaof the region. The compactness is measured in one embodiment by squaringthe length of the contour and dividing the result by four times the areaof the region multiplied by pi (Compactness=(contourlength)²/(4*area*π)).

The convexity measures the input regions to determine a measure of thelevel of convexity for a region. The convexity is measured in oneembodiment by dividing the original area of the region by the area ofthe convex hull (Convexity=(area of the region)/(area of the convexhull)).

The bulkiness and structure factor are features that can be derived fromelliptical parameters derived from the region. The region is convertedinto an elliptical measurements for radius A R_(a), radius B R_(b), andan area A, where R_(a) is ≧R_(b). An additional feature, anisometry, canbe calculated in one embodiment as the ratio of R_(a) to R_(b)(Anisometry=R_(a)/R_(b)). The bulkiness is calculated in one embodimentas pi multiplied by R_(a) and R_(b), divided by A(Bulkiness=π*R_(a)*R_(b)/A). The structure factor is calculated in oneembodiment as the Anisometry multiplied by the Bulkiness minus one(Structure Factor=(Anisometry*Bulkiness)−1).

(f) Feature Vector—Surface Features with Rotational Independence

Additional features can also be extracted from the image to reflect thesurface features of the pill, in one embodiment. FIG. 6 is a flowchartillustrating a method of extracting surface features from a pill imagein one embodiment. FIGS. 7-13 illustrate image processing techniquesused according to an embodiment for extracting surface features. FIG. 6is described herein with joint reference to FIGS. 7-13 to illustrate theimage processing. After starting the surface feature extraction 600, thepill image is accessed 601. The accessed image is converted to grayscaleas shown in FIG. 7. In the image shown in FIG. 5, a grayscale image ofthe pill is shown after smoothing the image received from a camera orother imaging device.

Next, using the outline of the pill, the pill edge is eroded and thepill border is replicated 602 outward to smooth the transition from thepill to the remaining portions of the pill image. The resulting erosionand replication is illustrated in FIG. 8. An anisotropic filter isapplied 603, which further smoothes the pill image and enhancescontrasting features in the pill. The pill image after the anisotropicfilter is shown in FIG. 9.

Next, a gray range filter is applied 604. The gray range filter convertsthe image such that the value of each pixel is equal to the rangedifference of gray values in a square of a given size around the pixel.The result emphasizes surface features in the image and is shown in FIG.10. In one embodiment, a feature representing the surface complexity iscaptured by integrating over the image shown in FIG. 10. The surfacecomplexity feature may act as a proxy for the amount of surfaceinformation available from the pill surface in that image and may beused for scoring pill classifications as described below. In otherembodiments, a measure of surface complexity may be obtained at anotherpoint in the surface-feature process.

Since the image could have been captured from a variety of angles,especially if the pill is rotating or otherwise in motion as the imagesor taken, the next processing step provides rotational independence tothe captured image by converting the image to polar coordinates 605 asshown in FIG. 11. By using a combination of polar coordinates andpolar-independent statistics, various pill rotations produce similarfeature vectors. The similar feature vectors are possible because thepolar transformation enables statistics calculated on rows of the polarimage which are independent of the pill rotation in the original image.The rotational independence allows reliable classification using a widevariety of pill images.

Next, the polar-coordinate image is subsampled 606 in one embodiment asshown by FIG. 12. The subsampling technique generalizes the data andreduces the likelihood the features produced by the same pill typesubstantially differ. Next, the subsampled image can be divided intorows as seen in FIG. 13. Each row 1301 is used to generate 607 a set ofrotation-independent surface features. Since each row is in polarcoordinates, each row represents a “ring” radially outward from thecenter of the original image. The features extracted from the row 1301are independent of pixel order. As a result, the features characterizeeach ring extending outwardly from the center of the original image. Theuse of polar coordinates and features measured independent of pixelorder provide a rotation-independent feature set. The features extractedfrom the row 1301 can include mean gray value, standard deviation ofgray values, entropy of gray values, and anisotropy of gray values. Inpractice, any characteristics could be used which provide measurementsof a row which is rotationally-independent. The features extracted fromthe subsampled polar image are appended to the feature vector.

(g) Feature Vector—Color Components

Further features are extracted to analyze the pill image's colorcomponents, as shown in FIG. 14. In one embodiment, using the originalcolor image, the color component calculation is started 1401. The colorimage is converted 1402 into a hue-saturation-value image. Next, atwo-dimensional histogram is created 1403 based on the hue andsaturation values for each pixel. Each bin in the two-dimensionalhistogram represents a particular combination of hue and saturation andincludes the pixels associated with that particular combination. Assuch, the “value” component of hue-saturation-value is discarded in thisembodiment and the histogram does not include the pixel brightness. Inan embodiment, the histogram is further normalized to enhance comparisonacross various histograms. Histogram statistics are calculated 1404,including center of mass coordinates and elliptical axes of the grayvalue moments. The hue-saturation histogram is next reduced 1405 inresolution. The reduction in resolution of the histogram is similar tothe subsampling technique used above with respect to surface features.The reduction in resolution reduces the dimensionality of the featurevector and thereby generalizes the data. Further statistics arecalculated 1406 on the reduced-resolution histogram. In one embodiment,the average gray values of each row and column are measured for thereduced-resolution histogram. The additional features extracted from thecolor histograms are appended to the feature vector.

(h) Combined Image Features

In addition to the individual image statistics, the feature vector foreach image is augmented in one embodiment with features representativeof the set of images taken for a pill. For example, certain pill typeshave different colors on different sides of a pill. For these pills, anindividual image may capture only one of the colors on the pill, and thepill type is prone to misclassification. Since other images of the pillmay include the further colors on the pill, including the color datafrom other images to the classification of the individual images improvepill classification of that individual image. For example, if a pill iswhite on one side and blue on the other side, an image of the white sideof the pill has a greater chance of being misclassified as a differentwhite pill. However, including information from images that also includethe blue side minimizes or eliminates this misclassification.

As such, features from the set of images of a pill are appended to thefeature vector for each image. The features which are appended from theset of images can include any features which may vary from differentangles of the image and have high discriminatory value. For example,these features can include the color-related features described above aswell as the size and shape features described above. With regard to sizeand shape features, an image of the short dimension of a pill could bemisclassified as a different, smaller pill whose long dimension issimilar. However, with the inclusion of the maximum long dimensionlength from another image in the feature vector, such amisclassification can be minimized or eliminated.

As a result of the preceding feature extraction techniques, a featurevector is extracted from the image taken of the pill. Next, the featurevector is used as an input to a classifier or plurality of classifiersto determine which pill the image depicts. Alternatively, the featurevector is used to train the classifiers for later pill identification.

(i) Pill Classification

The pill classifiers can be implemented using a variety of machinelearning techniques. For example, the machine learning techniques areimplemented in one embodiment using a support vector machine (SVM).Another embodiment uses a Gaussian mixture model. A further embodimentuses a neural network. The classifiers are typically implemented asmathematical functions which take as inputs a feature vector and produceas an output a pill type. For example, the classifier can output aBoolean value indicating a match or not a match, or it can output arange of values indicating a confidence score for a pill type.Typically, the classifier may be trained using a set of pill imagesidentified with a particular pill type, so that the classifier canidentify the pill type in a future image. The classifier scores are usedfor determining which of the pill types is identified in the pill image.

FIG. 15 illustrates a hierarchical classification methodology foridentifying the feature vectors for the pill images. Since there arethousands or tens of thousands, or more kinds of pills that may beidentified, a classifier that attempts to distinguish these individualpills in a single classifier may not produce satisfactory results. Inthis method, several classifiers are used in a hierarchical fashion todiscriminate between types of pills. The feature vector is initiallyprovided to a top-level classifier 1510. The top-level classifier 1510classifies the feature vector using only the color-based components ofthe feature vector. In this example, the feature vector of an imageincludes the color-based features from other pill images taken of thepill. The top-level classifier is configured to determine which clusterof pill types the feature vector belongs to. For example, when usingcolor, the pills may be grouped, or clustered into sets of pills basedon the color of the pill. In this case, the top-level classifier 1510determines the feature vector is a white pill. An intermediateclassifier 1520 is chosen that corresponds to white pills.

In this embodiment, the intermediate classifier 1520 furtherdistinguishes pills based on color, size, and shape components in asimilar manner to the top-level classifier. In this example, theintermediate-level classifier 1520 classifies the feature vector 1500 asan ivory, small, and oval pill. Thus, the intermediate-level classifier1520 may differentiate finer color distinctions relative to thetop-level classifier 1510. While this description of the classificationis helpful for comprehending the hierarchical concepts of pillclassification, in practice no such formal definition of the selectionof the next-level classifier is made. In particular, each level of pillin the hierarchy may be determined by clustering pills according to thefeatures that will be used at each classification level. That is, thetop-level color classifier 1510 is trained by clustering pill typesaccording to color feature vectors and trained to classify pills ascluster 1, cluster 2, and so forth. The “white” description is mademerely to further understanding of the hierarchical classification.Thus, for example, the intermediate-level classifiers 1520 may classifyfeature vectors among a set of bottom-level classifiers 1530 thatnominally include pills that are similar to the human eye, such as{ivory, small, oval} and {ivory, medium, oval}.

After identifying a bottom-level classifier 1530 corresponding to theclassification by the intermediate-level classifier 1520, thebottom-level classifier 1530 classifies the pill using the entirefeature vector. As such, the bottom-level classifier identifies the pillas a particular pill identification 1540. As shown in this example, thebottom-level classifier may also indicate a rank-order of pills, forexample indicating that the pill is most-likely the first pillidentification 1540 (Watson 349), but also the second most-likely pillidentification (Jones 150). Additional rank orders may be furthergenerated by the bottom-level classifier 1530. By using the hierarchicalclassification method, each level of classifier may focus on thediscriminatory features of each level. In addition, each classifier hasa limited number of classifications to make, rather than the thousandsor tens of thousands that would be used if a single classifier was used.The pill identification 1540 is thus determined for each feature vector.

(j) Classification Scoring

A method of pill identification using classifier scoring is shown inFIG. 16. In this embodiment, each classification result is used as a“vote” to determine which pill type to ultimately return as the pilltype for the pill images. One method of determining which pill type toreturn is shown in FIG. 16. The method used in this figure illustratesan embodiment where the classifier is a SVM which can return the“most-probable” classification and the “second-most” probableclassification, but does not return confidence scores.

At the starting point 1601, the plurality of images taken of the pillhave been processed by extracting feature vectors. Next, the classifiersare applied 1602 to each pill image to determine 1603 the most probableand second-most probably pill type for each image. The scoring methodproceeds by assigning scores 1604 to the classification results, such asby assigning a 1.0 score for the most-probable classification and a 0.5score for the second-most probable classification. The scores for themost-probable and second-most probable classification may be modified bya confidence value for the classifications. For a classifier that doesnot natively provide a confidence value, a confidence value may bedetermined by analyzing the results of the classifier. For a SVM, theSVM is composed of individual binary classifiers. When the SVMdetermines a particular pill is more probable than another pill, theindividual binary classifiers in the SVM have “voted” for themore-probable pill. Thus, the binary classification results from thebottom-level classifier 1530 may be accessed to determine the resultsfrom the individual binary classifiers in the bottom-level classifier1530. The disparity between the most-probable and second-most probablepill type identification may be used to adjust the scoring between theresults. If the second-most probable was distinguishable by fewer binaryclassifiers, the score for the second-most probable pill classificationis increased closer to the value of the most-probable pillidentification.

In one embodiment, after determining the scores for the pills of eachimage, the scores for each image may also be weighed relative to thesurface complexity of each image. As described above, the surfacecomplexity feature describes the extent of surface features and othercharacteristics on each pill image. Thus, the pill images that includeless surface feature complexity may have fewer distinguishingcharacteristics with which to identify an accurate pill type. Thus, thescores associated with pill images with reduced complexity are weightedto reduce the scoring from classification.

The scores are used to vote 1604 for each pill type to tally theclassification results for the pill images. The highest-scoring pilltype is the type identified as the pill type for the pill. An example ofthis pill classification system is shown in chart 1 below:

CHART 1 Pill Pill Pill Pill Type 1 Type 2 Type 3 Type 4 Image A 1.0 0.90 0 Image B 0.75 0 0 0 Image C 0.8 0 0.5 0 Image D 0.3 0 0 0.5 Total2.85 .9 0.5 0.5

Chart 1 shows a set of four images which have been classified accordingto four pill types. Each image is classified to identify the most- andsecond-most-likely pill type. The scores in this chart have beenadjusted to account for a confidence value and the relative surfacefeature complexity of each image. The confidence value of the pillclassification may be seen by comparing the values of each pill type foran image. For example, image A has a similar score for pill type 2because the classifier ranked pill types 1 and 2 similar, and inparticular may differ by only a few binary classifiers in thebottom-level classifier 1530. The surface feature complexity may be seenby comparing the maximum scores of each image. For example, image A hasa maximum score of 1.0, compared to the maximum score for image D of 0.5because image A has more a complex surface relative to image D and isaccorded more weight. The classification scores are tallied, and pilltype 1 is identified as the best-scoring pill type among the images.Pill type 1 is identified 1606 and reported as the pill type for thepill. In one embodiment, the total tally must reach a threshold beforethe pill is identified as a pill type. The use of a threshold reducesthe likelihood a pill which has a pill type which has not been trainedby the system is mistakenly classified as an existing pill type.

In another embodiment, the classifiers produce a confidence score foreach pill type rather than a “most likely” pill type. Using confidencescores, the system can also determine that no pill type is sufficientlyconfident for a match, and therefore the pill is not identified as aknown pill type.

(k) Pill Identification System

An embodiment of a pill identification system 1700 is provided in FIG.17. The pill identification system 1700 may be connected to the systemillustrated in FIG. 1 for capturing pill images. The pill images areprovided to the pill identification system 1700 and the resulting pillidentification determines the result bin 140 for placing a pill. Inother embodiments, the pill identification system 1700 may be separatelylocated from the system capturing pill images.

The pill identification system 1700 includes a pill photo database 1701,which includes raw photos of pills. The pill photo database receivesphotos taken by a camera and may store them indefinitely or for aspecified period. In one embodiment the pill photo database receives newpill photos and maintains the photos only during the pill identificationprocess. The feature extraction module 1702 extracts features from thepill photos as described above by extracting size, shape, color, andsurface features from the pill images and constructing feature vectors.The resulting feature vectors are stored in the feature database 1703.

The classifier training module 1704 trains a classifier or a set ofclassifiers based on the extracted pill features and the pill typedesignations. The pill type designations are used in conjunction withassociated feature vectors to train the classifiers. The trainedclassifier or classifier set is stored in a pill classifier database1705 to be used for classifying a new pill. In some embodiments, a pillidentification system contains a pre-configured set of classifiers, suchthat the pill identification system does not train classifiers, and theclassifiers are pre-configured. The classifiers can also be loaded froman external source, such as by transmitting classifiers from a networkor receiving them from an external non-volatile storage. Thus, anexternal system may provide updates to change the pills classified bythe database. These configurations may omit the classifier trainingmodule 1704.

Using the pill classifications associated with each image and featurevector, from the pill classifiers, a classifier voting module 1706provide a determination of an individual pill. The classifier votingmodule 1706 can weigh classification voting from a plurality of appliedphotos as provided above. The classifier voting module 1706 thenprovides the most-likely pill classification as the pill type to theuser or to control a gate 130 and diverter 150 as shown in FIG. 1.

The foregoing description of the embodiments of the invention has beenpresented for the purpose of illustration; it is not intended to beexhaustive or to limit the invention to the precise forms disclosed.Persons skilled in the relevant art can appreciate that manymodifications and variations are possible in light of the abovedisclosure.

Some portions of this description describe the embodiments of theinvention in terms of algorithms and symbolic representations ofoperations on information. These algorithmic descriptions andrepresentations are commonly used by those skilled in the dataprocessing arts to convey the substance of their work effectively toothers skilled in the art. These operations, while describedfunctionally, computationally, or logically, are understood to beimplemented by computer programs or equivalent electrical circuits,microcode, or the like. Furthermore, it has also proven convenient attimes, to refer to these arrangements of operations as modules, withoutloss of generality. The described operations and their associatedmodules may be embodied in software, firmware, hardware, or anycombinations thereof.

Any of the steps, operations, or processes described herein may beperformed or implemented with one or more hardware or software modules,alone or in combination with other devices. In one embodiment, asoftware module is implemented with a computer program productcomprising a computer-readable medium containing computer program code,which can be executed by a computer processor for performing any or allof the steps, operations, or processes described.

Embodiments of the invention may also relate to an apparatus forperforming the operations herein. This apparatus may be speciallyconstructed for the required purposes, and/or it may comprise ageneral-purpose computing device selectively activated or reconfiguredby a computer program stored in the computer. Such a computer programmay be stored in a non-transitory, tangible computer readable storagemedium, or any type of media suitable for storing electronicinstructions, which may be coupled to a computer system bus.Furthermore, any computing systems referred to in the specification mayinclude a single processor or may be architectures employing multipleprocessor designs for increased computing capability.

Embodiments of the invention may also relate to a product that isproduced by a computing process described herein. Such a product maycomprise information resulting from a computing process, where theinformation is stored on a non-transitory, tangible computer readablestorage medium and may include any embodiment of a computer programproduct or other data combination described herein.

Finally, the language used in the specification has been principallyselected for readability and instructional purposes, and it may not havebeen selected to delineate or circumscribe the inventive subject matter.It is therefore intended that the scope of the invention be limited notby this detailed description, but rather by any claims that issue on anapplication based hereon. Accordingly, the disclosure of the embodimentsof the invention is intended to be illustrative, but not limiting, ofthe scope of the invention, which is set forth in the following claims.

What is claimed is:
 1. A computer-implemented method for identifying apharmaceutical composition, the method comprising: accessing a set ofimages of the pharmaceutical composition, the set of images including: afirst image of the pharmaceutical composition from a first point ofview, and a second image of the pharmaceutical composition from a secondpoint of view different from the first point of view; extractingfeatures from each image of the pharmaceutical composition in the set ofimages; generating a feature vector associated with each image of thepharmaceutical composition that includes the features extracted for thatimage and one or more features appended from another image in the set ofimages; applying at least one classifier to each of the feature vectorsto determine at least one classification associated with each image; anddetermining a pill identification by accumulating the at least oneclassification for each image.
 2. The computer-implemented method ofclaim 1, further comprising: receiving a pill verification requestspecifying a desired pill type; determining whether the pillidentification matches the desired pill type; and reporting thedetermination of whether the pill identification matches the desiredpill type.
 3. The computer-implemented method of claim 2, the methodfurther comprising directing the pharmaceutical composition to anaccepted bin responsive to the pill identification matching the desiredpill type and directing the pharmaceutical composition to a rejected binresponsive to a failure to match.
 4. The computer-implemented method ofclaim 1, wherein the extracted features further include color, shape,size, and surface features.
 5. The computer-implemented method of claim4, wherein the feature extraction of each image includes performing thesteps of: identifying a pill outline of the image; applying an erosionfilter to the pill outline of the image; replicating the pill outline tothe exterior of the image after the erosion filter is applied; andapplying an anisotropic filter to the image after the pill outline isreplicated.
 6. The computer-implemented method of claim 1, wherein thefeatures include a measure of surface features of the pharmaceuticalcomposition in the image, wherein the measure of surface features isdetermined for each image by performing the steps of: converting theimage to a polar coordinate image; subsampling the polar coordinateimage; and extracting the measure of surface features from thesubsampled polar coordinate image.
 7. The computer-implemented method ofclaim 1, wherein the appended one or more features in the feature vectorassociated with each image of the pharmaceutical composition includes atleast one of shape, color, and size-related features extracted from theother image.
 8. The computer-implemented method of claim 7, wherein theappended features extracted from the other image are sorted withfeatures from the image of the pharmaceutical composition prior toappending the sorted features to the feature vector.
 9. Thecomputer-implemented method of claim 1, wherein the at least oneclassifier comprises a set of hierarchical classifiers comprising atleast one top-level classifier and at least one bottom-level classifier.10. The computer-implemented method of claim 9, wherein the featurevectors include color features and the top-level classifier performsclassification based on the color features in the feature vector. 11.The computer-implemented method of claim 9, wherein the feature vectorsinclude size and shape features and the set of hierarchical classifiersfurther comprise at least one intermediate classifier that performsclassification based on the size and shape features in the featurevector.
 12. The computer-implemented method of claim 9, wherein the atleast one bottom-level classifier performs classification based onsurface features.
 13. The computer-implemented method of claim 1,wherein determining the pill identification by accumulating the at leastone classification for each image comprises determining a weighting foreach image classification based on a confidence level of the respectiveclassifications.
 14. The computer-implemented method of claim 1, whereinthe feature vector associated with each image includes a surfacecomplexity feature, and wherein determining the pill identification byaccumulating the at least one classification for each image comprisesweighting the at least one classification for each image based on a therelative surface complexity feature of the feature vector associatedwith the image.
 15. A computer-implemented method for identifying apharmaceutical composition, the method comprising: accessing a set ofimages of the pharmaceutical composition, the set of images including: afirst image of the pharmaceutical composition from a first point ofview, and a second image of the pharmaceutical composition from a secondpoint of view different from the first point of view; developing a setof features for each image, wherein the features are developed for eachimage by: identifying, in the image, an outline of the pharmaceuticalcomposition; calculating, from the image, a characterization of theimage based at least in part on the identified outline; reducing theresolution of the characterization of the image; extracting, from thereduced resolution characterization of the image, features indicative ofsurface features of the image which comprise at least a portion of thefeatures developed for the image; appending to the set of features oneor more features from another image in the set of images; and providingthe set of features for each image to a classification module forclassifier training for pharmaceutical composition identification. 16.The computer-implemented method of claim 15, wherein the extracting stepfurther extracts color, size, and shape features from the image.
 17. Thecomputer-implemented method of claim 15, wherein the outline of thepharmaceutical composition is identified using an edge-detectionalgorithm.
 18. The computer-implemented method of claim 15, wherein theoutline of the pharmaceutical composition is identified using agrayscale segmentation algorithm.
 19. The computer-implemented method ofclaim 15, further comprising training a set of hierarchical classifiersbased at least in part on the set of features for each image.